An LPR (License Plate Recognition) system is a surveillance method that typically utilizes OCR (Optical Character Recognition) on images to read vehicle license plates and other identifying information. Some LPR systems utilize existing closed-circuit televisions or road-rule enforcement cameras, or cameras designed specifically for the surveillance task at hand. LPR systems are employed by various tolling agencies and companies as a method of electronic toll collection on, for example, pay-per-use roads and cataloging the movements of traffic or individuals.
Some LPR systems have been developed, which are composed of several modules, the first of which involves license plate localization where regions of the input image are identified as potentially containing a license plate. These sub images are referred to as ‘Region of Interest’ (ROI) images. By design, some LPR systems assume that at least one license plate exists in the input image and thus the processing is biased to generate many candidate ROI's to ensure that a license plate is found. This approach would be typical of, for instance, a tolling application wherein images are captured as a vehicle travels through a toll plaza or overhead gantry. Each generated ROI is passed to character segmentation, then OCR, and finally State ID subsystems. An ROI can be rejected at any step of the process.
One of the problems with some current LPR systems is the rejection of valid license plate(s). To illustrate this problem, consider several rejection examples. In one rejection scenario, segmentation may return less than 4 characters. In another rejection scenario, too many gaps may exist between the segmented characters. In still another rejection example, the OCR operation may return a low confidence for all of the characters. Additionally, in some situations, state identification via the state identification module 20 may fail to reach a conclusion.
In this manner, ROI's that do not contain a license plate are likely to be discarded by subsystems downstream from localization. In addition, ROI's that contain a license plate can also be discarded if conditions such as these aren't met as part of the process flow. This highly selective behavior helps the automated OCR solution reduce the number of mistakes that it makes. Given the market requirements for highly accurate OCR (99% or better), LPR systems or engines tend to err on the conservative side, since problematic or difficult to read license plates are risky to evaluate with high confidence.
Since they are highly optimized to meet the demanding market requirements for accuracy, LPR solutions are not usually very good at determining with high accuracy whether there is (or is not) a license plate in a given image. Rather, the LPR system is typically quite good at determining whether there is a license plate in the image that is easy to OCR with high accuracy. This leads to a larger than desired pool of images that are sent for human review (at added cost). For the reasons outlined above, a separate method is required that is optimized for this particular problem. If an LPR returns a highly confident conclusion, then we can assume that a license plate exists, but not the other way around.
In cases where an LPR engine does not return a conclusion or returns a conclusion with confidence below a predetermined accuracy threshold, the images are forwarded to a human for review. A human can take various measures to determine the license plate code and state including inferring various details in cases where the license plate is partially occluded. When a human successfully determines the plate code and state, the driver is then charged a toll (or a fine for a photo-enforcement application). For some fraction of the human reviewed images, there was in fact no readable license plate present in the image. Here the investment in time and cost for a human review is wasted. This can occur when there are image capture triggering problems, image quality problems, occlusions from other objects in the image, improperly mounted plates, or for cases where there is actually no plate physically on the vehicle. The actual absence of a license plate is a particular problem in California where by law VCS 4456, drivers of new vehicles have up to three months (recently reduced from 6) to register the vehicle with the state. Given this, some tolling operations in Los Angeles, for example, are seeing approximately 15% of traffic without license plates, which leads to a corresponding human review expense.
Tolling images, which have fundamental readability problems are first labeled with the type of problem and then are rejected from billing. The process of describing why an image is not readable is called the ‘image reject process’ (IRP). Missing license plates are one category of ‘rejects’.
Other methods exist for attempting to detect objects of interest in images. An example is that of a face detector, which has become commonplace in digital cameras today. In fact, as part of any LPR system, some form of license plate detector will likely be used to identify candidate regions of interest where further processing should be applied. In some situations, an image-based classifier (license plate detector) based on SMQT features and the SNoW classifier to identify candidate ROIs may be employed.
For any object detection/classification problem, there is a tradeoff between what are called “missed detections” and “false alarms”. In fact, common practice in this space is to develop a receiver operating characteristic (ROC) curve that enables one to select the desired operating point—i.e. the “best” tradeoff between false alarms and missed detections for the application of interest.
For the application of screening images from the human review pool, any image that is not sent for human review means that any associated toll cannot be collected. Thus, a “false alarm” by the automated detection method—identifying an image as not having a plate when in fact it does—translates directly into lost revenue. So, in order to be a viable option for automatically screening images from the human review pool, an object detection method has to meet extremely aggressive performance requirements, for example, a maximum false alarm rate of 0.5%.
Unfortunately standard object detection methods, including the SMQT/SNoW classifier used by LPR systems, do not provide sufficient performance for accurately identifying whether there is a license plate in an image or not. Existing classification/detection approaches provided insufficient yield (accurately detected images that don't contain license plates) at the required false alarm rate. The present invention addresses this gap by leveraging a specific combination of image features and classification methods as part of an overall LPR process flow that achieves the required performance targets.